{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Data Acquisition, Web Scraping and Web APIs *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "* [Data Acquisition, Web Scraping and Web APIs *](#Lecture-5---Data-Acquisition,-Web-Scraping-and-Web-APIs-*)\n",
    "\t* &nbsp;\n",
    "\t\t* [Content](#Content)\n",
    "\t\t* [Learning Outcomes](#Learning-Outcomes)\n",
    "* [Data Acquisition](#Data-Acquisition)\n",
    "* [1. Web scraping](#1.-Web-scraping)\n",
    "\t* [HTML](#HTML)\n",
    "\t\t* &nbsp;\n",
    "\t\t\t* [What is HTML?](#What-is-HTML?)\n",
    "\t* [Intro to Web Scraping](#Intro-to-Web-Scraping)\n",
    "\t\t* [--- WARNING ---](#----WARNING----)\n",
    "\t* [2. Web APIs](#2.-Web-APIs)\n",
    "\t\t* [REST](#REST)\n",
    "\t\t* [JSON](#JSON)\n",
    "\t\t* [Forming an API query](#Forming-an-API-query)\n",
    "\t* [Current International Space Station Details](#Current-International-Space-Station-Details)\n",
    "\t* [Dedicated API Wrapper Modules](#Proprietary-API-Wrapper-Modules)\n",
    "\t* [API Repositories and Market Places](#API-Repositories-and-Market-Places)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "* Some material on web scraping and usage of APIs adapted from Kevin Markham's data science courses at https://github.com/justmarkham"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Data gathering via web scraping\n",
    "2. HTML basics\n",
    "3. Data gathering via web APIs\n",
    "4. JSON file format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Outcomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of this lecture, you should be able to:\n",
    "\n",
    "* list the different dynamic sources of data\n",
    "* explain what HTML is and its basic structure\n",
    "* make HTTP requests using python\n",
    "* traverse the HTML document tree\n",
    "* perform web scraping at an introductory level\n",
    "* describe and process the JSON file format\n",
    "* perform rudimentary data acquisition using Web APIs\n",
    "\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Acquisition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we have looked at how we can acquire data from pre-prepared Excel and text files in the CSV format. We also saw how we can use pandas clipboard facility to paste and build data frames. \n",
    "\n",
    "We also experienced that much of the data does not come in tidy formats that are prepared and ready for data analysis. For this we learned a number of techniques that help us to wrangle and tidy our data into shape. \n",
    "\n",
    "Now we are going to look at two additional sources of data that are dynamic and will require the combination of all the techniques we learned previously, such as wrangling, merging, aggregation, as well as some new skills. \n",
    "\n",
    "It is becoming common these days that data is acquired from multiple sources and merged into a single dataset. The data sources that are increasingly becoming the backbone of many analytics and information systems are web based.\n",
    "\n",
    "This section considers how data can be read (scraped) from web pages (HTML documents), and how data can be retrieved from web servers using their application program interfaces (APIs)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Web scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often when we need to acquire data, web pages are a great resource to turn to. Many websites make data available on their web pages for viewing in a browser, but do not make it conveniently downloadable as an easily machine-readable format like JSON, CSV, or XML. Because of this, we sometimes need to employ web scraping techniques.\n",
    "\n",
    "The term \"web scraping\" refers to an application or script that processes HTML pages. This is done in order to extract data embedded in HTML for manipulation. \n",
    "\n",
    "Web scraping applications in effect simulate a person viewing a website with a browser.\n",
    "\n",
    "Our task then becomes writing scripts that can traverse the structure of HTML documents and locate the particular piece of data we need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What is HTML?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HTML is a markup language (not a programming language) for describing web documents (web pages).\n",
    "\n",
    "    HTML stands for Hyper Text Markup Language\n",
    "    A markup language is a set of markup tags\n",
    "    HTML documents are described by HTML tags\n",
    "    Each HTML tag describes different document content\n",
    "\n",
    "HTML pages consist of elements. Elements are marked up by tags, and the tags may have attributes inside them which describe how the content should be rendered by web browsers. The initial tag specifies the type of the document so that the browsers render the content correctly.\n",
    "\n",
    "Please refer to http://www.w3schools.com/html/html_intro.asp for an introduction to HTML."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<!DOCTYPE html>\n",
    "<html>\n",
    "  <head>\n",
    "    <title>This is a title</title>\n",
    "  </head>\n",
    "  <body >\n",
    "    <p>Hello world!</p>\n",
    "  </body>\n",
    "</html>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The examples below will show how we can perform web scraping on HTML pages using a Python package called `BeautifulSoup`. \n",
    "\n",
    "BeautifulSoup is an HTML/XML parser for Python that can turn markup text into a parse tree, that can then be traversed more easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import HTML, IFrame\n",
    "IFrame(\"http://www.crummy.com/software/BeautifulSoup/bs4/doc/\", width=1100, height=500)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BeautifulSoup provides a simplified, idiomatic way of navigating, searching, and modifying parse tree generated by HTML and XML.\n",
    "\n",
    "More info on BeautifulSoup http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html\n",
    "\n",
    "Good examples of how this is done can be found in : http://www.gregreda.com/2013/03/03/web-scraping-101-with-python/ and http://blog.miguelgrinberg.com/post/easy-web-scraping-with-python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro to Web Scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to begin with a toy example first using the simple html page created below:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<!doctype html>\n",
    "<html lang=\"en\">\n",
    "<head>\n",
    "  <title>Teo's Webpage</title>\n",
    "</head>\n",
    "\n",
    "<body>\n",
    "  <h1>Teo's Webpage</h1>\n",
    "  <p id=\"intro\">My name is Teo.  I find web scraping interesting.</p>\n",
    "  <p id=\"background\">I live in Auckland and completed my PhD at Massey University in Computer Science, while studying the field of machine learning.</p>\n",
    "  <p id=\"current\">I currently work as a lecturer in Information Technology.</p>\n",
    "  \n",
    "  <h3>My Interests</h3>\n",
    "  <ul>\n",
    "      <li id=\"my favorite\">Data Science and Machine Learning</li>\n",
    "      <li class=\"hobby\">Tennis</li>\n",
    "      <li class=\"hobby\">Reading</li>\n",
    "      <li class=\"hobby\">Travelling</li>\n",
    "      <li class=\"hobby\">Running</li>\n",
    "  </ul>\n",
    "</body>\n",
    "</html>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "import requests                 # How Python gets the webpages\n",
    "from bs4 import BeautifulSoup   # Creates structured, searchable object\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import json\n",
    "#import prettyprint as pp\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pylab import rcParams\n",
    "\n",
    "rcParams['figure.figsize'] = 15, 10\n",
    "rcParams['font.size'] = 20\n",
    "rcParams['axes.facecolor'] = 'white'\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, let's read the toy webpage as a string - this is what happens initially when you scrape any webpage\n",
    "html_doc = \"\"\"\n",
    "<!doctype html>\n",
    "<html lang=\"en\">\n",
    "<head>\n",
    "  <title>Teo's Webpage</title>\n",
    "</head>\n",
    "\n",
    "<body>\n",
    "  <h1>Teo's Webpage</h1>\n",
    "  <p id=\"intro\">My name is Teo.  I find web scraping interesting.</p>\n",
    "  <p id=\"background\">I live in Auckland and completed my PhD at Massey University in Computer Science, while studying the field of machine learning.</p>\n",
    "  <p id=\"current\">I currently work as a lecturer in Information Technology.</p>\n",
    "  \n",
    "  <h3>My Interests</h3>\n",
    "  <ul>\n",
    "      <li id=\"my favorite\">Data Science and Machine Learning</li>\n",
    "      <li class=\"hobby\">Tennis</li>\n",
    "      <li class=\"hobby\">Reading</li>\n",
    "      <li class=\"hobby\">Travelling</li>\n",
    "      <li class=\"hobby\">Running</li>\n",
    "  </ul>\n",
    "</body>\n",
    "</html>\n",
    "\"\"\"\n",
    "type(html_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Beautiful soup allows us to create structure from the html elements, and to traverse it\n",
    "page = BeautifulSoup(html_doc, \"lxml\")\n",
    "print(type(page))\n",
    "page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The most useful methods in a Beautiful Soup object are \"find\" and \"findAll\".\n",
    "# \"find\" takes several parameters, the most important are \"name\" and \"attrs\".\n",
    "# name will help us find the type of an element\n",
    "# Let's target \"name\".\n",
    "page.find(name='body') # Finds the 'body' tag and everything inside of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "body = page.find(name='body')\n",
    "type(body) #element.Tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above result tells us that 'body' element was found in the HTML page, and it tells us what object type it is. When the find fails, then this is what we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "body = page.find(name='bodyyy')\n",
    "type(body) #element.Tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see its content below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "body = page.find(name='body')\n",
    "body.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can recursively search for other elements inside the returned result as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h1 = body.find(name='h1') # Find the 'h1' element inside of the 'body' tag\n",
    "h1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h1.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h1.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how we can access the entire element or just the content. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's find the 'p' elements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = page.find(name='p')\n",
    "# This only finds one.  This is where 'findAll' comes in.\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also do a search of all instances of an element:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_p = page.findAll(name='p')\n",
    "print(all_p)\n",
    "type(all_p) # Result sets are a lot like Python lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access specific element with index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(all_p[0])\n",
    "print(all_p[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iterable like  list\n",
    "for one_p in all_p:\n",
    "    print(one_p.text) # Print text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access specific attribute of a tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_p[0] # Specific element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_p[0]['id'] # Specific attribute value of a specific element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at 'attrs'. Beautiful soup also allows us to locate elements with specific attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "page.find(name='p', attrs={\"id\":\"intro\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "page.find(name='p', attrs={\"id\":\"background\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = page.find(name='p', attrs={\"id\":\"current\"})\n",
    "result.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again we can also do a search of all instances of an element and name of a class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "page.findAll(\"li\", \"hobby\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Extract the 'h3' element from Teo's webpage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Extract Teo's hobbies from the html_doc.  Print out the text of the hobby. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Extract Teo's hobby that has the id \"my favorite\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will illustrate this concept further on an NZ Economics (which does not forbid web scraping) website called https://tradingeconomics.com/, where will will attempt to scrape data describing various economic indices.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try and pull the exchange rate from the Euro to the USD.\n",
    "\n",
    "In order to find where the price is situated in the HTML document, we must look at the document's source code. By right clicking on a page in a browser, an option should be displayed allowing you to view the source.\n",
    "\n",
    "We must inspect the source so that we can find the element that houses this value. We can then use the python's BeautifulSoup package to **read and traverse through the HTML element tree** in order to extract the data that we want.\n",
    "\n",
    "There are three basic steps to scraping a single page:\n",
    "\n",
    "    1. Get (request) the page\n",
    "    2. Parse the page content (read and interpret the document structure)\n",
    "    3. Search through the content of interest\n",
    "\n",
    "\n",
    "Below is the example of a script that will access and display the population value for NZ:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we first need to make some extra imports\n",
    "import json\n",
    "from time import sleep\n",
    "from datetime import datetime\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**STEP 1: GET** Access the page and read it into the beautiful soup object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://tradingeconomics.com/\"\n",
    "response = requests.get(url) \n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### --- WARNING --- "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ALWAYS FIRST MAKE SURE THAT THE RESPONSE IS 200 - OTHERWISE YOU MIGHT HAVE AN ERROR, IN WHICH CASE YOU'D BE BEST TO STOP AND NOT TRY TO PROCESS THE DOCUMENT, SINCE THERE WILL BE NOTHING TO PROCESS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "page = response.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "page[:10000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**STEP 2: PARSE** Create a BeautifulSoup object that reads and parses the HTML page into a format that we can search and traverse."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "from urllib.request import urlopen\n",
    "url = \"https://tradingeconomics.com/\"\n",
    "page = urlopen(url)\n",
    "html = page.read().decode(\"utf-8\")\n",
    "scraping = BeautifulSoup(html, \"html.parser\")\n",
    "scraping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scraping = BeautifulSoup(page, \"lxml\") \n",
    "scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can search for a given tag, id or class name.\n",
    "\n",
    "**STEP 3: SEARCH** Search through the page for 'tr' (table row) type tags looking for the attribute 'data-symbol' with the 'EURUSD:CUR value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scraping = BeautifulSoup(page, \"html\") \n",
    "element1 = scraping.find( attrs={'data-symbol' : 'EURUSD:CUR'})\n",
    "element1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result needs more filtering in order to get to the target value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "element2 = element1.find( attrs={'id' : 'p'})\n",
    "element2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now arrived at the element holding the value and we need to extract the text and cast it to a float:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "float(element2.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are differnt ways of honing in on the data that we need from this, but one option could be to perform another search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "float(scraping.findAll( attrs={'id' : 'p'})[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it turns out, there are multiple tags in the document with this tag-name combination. "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "ATTRIBUTES\n",
    "\n",
    "Classes are NOT unique\n",
    "\n",
    "    You can use the same class on multiple elements.\n",
    "    You can use multiple classes on the same element.\n",
    "\n",
    "ID's are unique and elements with ID's are easy to extract\n",
    "\n",
    "    Each element can have only one ID\n",
    "    Each page can have only one element with that ID\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** From the same website, navigate to the appropriate webpage and make the changes required to scrape the USD to Singapore dollar exchange rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#step 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#step 2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# step 3\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also read in entire HTML tables into dataframe objects:\n",
    "\n",
    "Here is an example of how to read in the ASCII table from a wikipedia page: https://en.wikipedia.org/wiki/ASCII"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#step 1\n",
    "url = \"https://en.wikipedia.org/wiki/ASCII\"\n",
    "response = requests.get(url)\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "scraping_html_table_EQ = BeautifulSoup(response.content, \"lxml\") \n",
    "scraping_html_table_EQ = scraping_html_table_EQ.find_all(\"table\", \"wikitable\")\n",
    "df = pd.read_html(str(scraping_html_table_EQ))\n",
    "first_table_df = df[0]\n",
    "first_table_df.columns = first_table_df.iloc[0]\n",
    "first_table_df = first_table_df.iloc[2:]\n",
    "first_table_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Read the second table from the same webpage into a data frame object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#step 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Return to the Tradingeconomics website and scrape all the Australia and NZ stock exchange figures. Tidy the dataframe and plot the YTD values as a bar graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Web APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Web servers serve out web pages in the HTML format as they are requested by users. Web servers are also capable of providing data that is not formatted in HTML. \n",
    "\n",
    "These web servers provide public (and private) APIs through which users can interact, construct queries that the web servers understand, and receive data from them. \n",
    "\n",
    "Depending on who owns them, web servers will have different APIs. They usually provide developer help pages that demonstrate how they work and how queries can be constructed using HTTP which the servers understand.\n",
    "\n",
    "Many websites have public APIs providing data feeds via JSON or some other common formats. We will consider only **JSON** as it is becoming a standard, and is conveniently, virtually identical to python's dictionaries in its syntax. \n",
    "\n",
    "Increasingly though, in order to access these APIs we must register for API Keys. They are **credentials**. Some of them are free and simply require that an account be created with a given website, while others must be purchased and have limits on the amount of data that can be pulled.\n",
    "\n",
    "There are a number of ways to access these APIs. **REST** (Representational State Transfer) is becoming the most common mechanism and often uses **JSON** as the format for transmitting data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JSON (short for JavaScript Object Notation) has become one of the standard formats\n",
    "for sending data by HTTP request between web servers and browsers and other applications. \n",
    "\n",
    "It is a much more flexible data format than a tabular text form like CSV. \n",
    "\n",
    "Here is an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#In Python triple-quoted strings allow us to include strings that have escape chars in it.\n",
    "obj = \"\"\"\n",
    "{\"name\": \"Massey University\",\n",
    "\"campuses_NZ\": [\"Albany\", \"Palmerston North\", \"Wellington\"],\n",
    "\"campuses_international\": null,\n",
    "\"colleges\": [{\"name\": \"Sciences\", \"degrees\": 10, \"majors\": 30},\n",
    "{\"name\": \"Business\", \"degrees\": 8, \"majors\": 25}]\n",
    "}\n",
    "\"\"\"\n",
    "obj\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JSON is very nearly valid Python code with the exception of its null value `null` and\n",
    "some other nuances (such as disallowing trailing commas at the end of lists). The basic\n",
    "types are objects (dicts), arrays (lists), strings, numbers, booleans, and nulls. \n",
    "\n",
    "**All of the keys in an object must be strings**. There are several Python libraries for reading and\n",
    "writing JSON data. We will use `json` here as it is built into the Python standard library. \n",
    "\n",
    "To convert (deserialize) a JSON string from above to an equivalent Python object (`dict`), use `json.loads`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = json.loads(obj)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`json.dumps` on the other hand converts a Python object back to JSON:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "as_json = json.dumps(result)\n",
    "as_json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How you convert a JSON object or list of objects to a DataFrame or some other data\n",
    "structure for analysis will be up to you. Conveniently, you can pass a list of JSON objects\n",
    "to the DataFrame constructor and select a subset of the data fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "massey_colleges = pd.DataFrame(result['colleges'], columns=['name', 'degrees'])\n",
    "massey_colleges"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert a data frame back to a JSON object with the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "massey_colleges.to_json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### REST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**REST is a lightweight mechanism that is protocol independent, but often sits on top of the HTTP protocol** which enables applications to exchange data with severs. \n",
    "\n",
    "A combination of HTTP requests, together with valid REST queries can easily be constructed from Python. One easy-to-use method is through the `requests` package (http://docs.python-requests.org).\n",
    "\n",
    "Previously, using Web Services and SOAP would result in queries like:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<?xml version=\"1.0\"?>\n",
    "<soap:Envelope\n",
    "xmlns:soap=\"http://www.w3.org/2001/12/soap-envelope\"\n",
    "soap:encodingStyle=\"http://www.w3.org/2001/12/soap-encoding\">\n",
    " <soap:body pb=\"http://www.whitepages.com/phonebook\">\n",
    "  <pb:GetUserDetails>\n",
    "   <pb:UserID>12345678</pb:UserID>\n",
    "  </pb:GetUserDetails>\n",
    " </soap:Body>\n",
    "</soap:Envelope>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using REST, such clumsy queries can be transformed into simple HTTP requests of a format (1) like:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "http://www.whitepages.com/phonebook/UserDetails/12345678"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or alternatively, passing arguments using format (2) as follows:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "http://www.whitepages.com/phonebook?UserDetails=12345678"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are slight differences in what you can expect from the two formats. Format 1 (**path segment parameter**) will return a 404 error when the parameter value does not correspond to an existing resource. \n",
    "\n",
    "Format 2 uses **optional parameters**. Instead of en error, this format will return an empty list when the parameter is not found in the query result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples of Forming API Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Science Toolkit (http://dstk.britecorepro.com/) \n",
    "\n",
    "The Data Science Toolkit provides free APIs for accessing a range of services.\n",
    "\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Google-style Geocoder \n",
    "\n",
    "Interfaces with Google's geocoding API. Provides the latitude and longitude of an address. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Albany Library,30 Kell Dr,Albany,Auckland,New Zealand\n",
    "response = requests.get(\"http://dstk.britecorepro.com/maps/api/geocode/json?sensor=false&address=Albany Library,30+Kell+Dr,Albany,+Auckland,+New+Zealand\")\n",
    "print(response)\n",
    "json.loads(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Extract the latitude and longitude of your current residential address and check it on Google maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### IP Address to Coordinates\n",
    "\n",
    "This API takes either a single numerical IP address, a comma-separated list, or a JSON-encoded array of addresses, and returns a JSON object with a key for every IP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(\"http://dstk.britecorepro.com//ip2coordinates/130.12.1.34\")\n",
    "print(response)\n",
    "json.loads(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Find out your current IP address (https://www.whatsmyip.org/ ), then extract the latitude and longitude check it on Google maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CoinDesk\n",
    "\n",
    "CoinDesk (https://www.coindesk.com/api) provides a simple API to make its Bitcoin Price Index (BPI) data programmatically available to others. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(\"https://api.coindesk.com/v1/bpi/currentprice.json\")\n",
    "print(response)\n",
    "json.loads(response.content)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Modify the query above in order to extract the current price of BitCoin in Singapore (SGD) dollars."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Modify the query above in order to extract the historic value of BitCoin in NZD dollars from 2013 to today. Extract the values from the json object and plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenRates\n",
    "\n",
    "OpenRates (http://www.openrates.io/) delivers and up-to-date exchange rate data for 32 world currencies in JSON format. All currency data is sourced from the European Central Bank. The OpenRates API also offers historical exchange rates back to 1999. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#base currency is the Euro\n",
    "response = requests.get(\"http://api.openrates.io/latest\")\n",
    "print(response)\n",
    "json.loads(response.content)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Modify the above query in order to extract the latest exchange rates showing how many NZD dollars a SGD dollar buys. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Make API queries that will enable you to quantify the percentage change in the number of Australian dollars a NZ dollar could buy between today and exactly one year ago."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RestCountries\n",
    "\n",
    "REST Countries (https://restcountries.eu/) provides high level information about a comprehensive set of countries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(\"https://restcountries.eu/rest/v2/all\")\n",
    "print(response)\n",
    "json.loads(response.content)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An API call to RestCountries can return a list of countries that a target country shares a border with, as well as the GINI coefficient that signifies income inequalities.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Make a set of API quesries which returns a list of countries with which Venezuela shares a border, then plot the GINI coefficient for all these countries including Venezuela. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Current International Space Station Details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://open-notify.org/Open-Notify-API/\n",
    "\n",
    "Examples below are taken from: https://www.dataquest.io/blog/python-api-tutorial/\n",
    "\n",
    "Below is an example of querying the ISS in order to find out the location of the space station."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a get request to get the latest position of the international space station from the opennotify api.\n",
    "response = requests.get(\"http://api.open-notify.org/iss-now.json\")\n",
    "\n",
    "# Print the status code of the response.\n",
    "print(response.status_code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The example API query below returns a list of upcoming ISS passes for a particular location formatted as JSON."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pprint as pp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Set up the parameters we want to pass to the API.\n",
    "# This is the latitude and longitude of New York City.\n",
    "parameters = {\"lat\": 40.71, \"lon\": -74}\n",
    "\n",
    "# Make a get request with the parameters.\n",
    "response = requests.get(\"http://api.open-notify.org/iss-pass.json\", params=parameters)\n",
    "\n",
    "# Print the content of the response (the data the server returned)\n",
    "pp.pprint(json.loads(response.content))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This gets the same data as the command above\n",
    "response = requests.get(\"http://api.open-notify.org/iss-pass.json?lat=40.71&lon=-74\")\n",
    "pp.pprint(json.loads(response.content))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Find the latitude and longitude for Singapore and query the API for ISS as to when the predicted flybys over Singapore will be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Iterate through the results of the above query and convert the flyover time from Epoch Time to local time expressed in a readable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this API, we can also find out programatically how many astronauts are currently in the ISS and who they are: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Get the response from the API endpoint.\n",
    "response = requests.get(\"http://api.open-notify.org/astros.json\")\n",
    "data = response.json()\n",
    "\n",
    "# 9 people are currently in space.\n",
    "print(data[\"number\"])\n",
    "pp.pprint(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GeoNet API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.geonet.org.nz/\n",
    "\n",
    "> GeoNet is the result of a partnership between the Earthquake Commission (EQC), GNS Science, and Land Information New Zealand (LINZ). The GeoNet project was established in 2001 to build and operate a modern geological hazard monitoring system in New Zealand. It comprises a network of geophysical instruments, automated software applications and skilled staff to detect, analyse and respond to earthquakes, volcanic activity, large landslides, tsunami and the slow deformation that precedes large earthquakes.\n",
    "\n",
    "\n",
    "GeoNet has an API from which latest seismic activity in the NZ region can be accessed: https://api.geonet.org.nz/\n",
    "\n",
    "\n",
    "Here is an example of how to query their API and extract all recent seismic activity that was at or above the Modified Mercalli Intensity scale (MMI)(https://en.wikipedia.org/wiki/Mercalli_intensity_scale)\n",
    "\n",
    "See https://api.geonet.org.nz/quake?MMI=3 for an example of the returned JSON object format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response = requests.get(\"https://api.geonet.org.nz/quake?MMI=2\")\n",
    "#print(response.content)\n",
    "res = json.loads(response.content)\n",
    "pp.pprint(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now search through the JSON object and extract all the seismic activity in the 'Wellington' region for example and print out the magnitude of the quake.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for i in range(len(res['features'])):\n",
    "    if res['features'][i]['properties']['locality'].find('Wellington') > -1:\n",
    "        print(res['features'][i]['properties']['locality'], ' - Richter Scale Magnitude: ', res['features'][i]['properties']['magnitude'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(res['features'])):\n",
    "    if res['features'][i]['properties']['locality'].find('Seddon') > -1:\n",
    "        print(res['features'][i]['properties']['locality'], ' - Richter Scale Magnitude: ', res['features'][i]['properties']['magnitude'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(res['features'])):\n",
    "    if res['features'][i]['properties']['locality'].find('Gisborne') > -1:\n",
    "        print(res['features'][i]['properties']['locality'], ' - Richter Scale Magnitude: ', res['features'][i]['properties']['magnitude'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Modify the above code in order to generate a new query to return all seismic activity that was at or above 2 MMI. Then print all the results for activity in the vicinity of Gisborne , listing both the date of the quake and the depth.a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** https://api.geonet.org.nz/quake/stats returns the stats of all seismic activity in NZ over the last year. Generate the query that returns this JSON object and then plot the NZ daily seismic activity in NZ over the last year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  World Bank API\n",
    "\n",
    "World Bank APIs provide access to various types of data and databases:\n",
    "\n",
    "    The Indicators API provides programmatic access to time series development data and metadata. Most of the articles in this section are devoted to the Indicators API.\n",
    "\n",
    "    The Data Catalog API provides information about the thousands of development-relevant datasets available through the World Bank Data Catalog. \n",
    "\n",
    "    The Projects API provides access to World Bank operations data, i.e., active, pipeline and closed projects implemented in countries and around the world. \n",
    "\n",
    "    The Finances API provides programmatic access to World Bank financial data (loans, credits, financial statements, etc) delivered on the World Bank Finances platform.\n",
    "\n",
    "    The Climate Data API provides access to historical and modelled climate data from the Climate Knowledge Portal. \n",
    "\n",
    "\n",
    "> Source: https://datahelpdesk.worldbank.org/knowledgebase/topics/125589-developer-information"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "The World Bank API enpoint is:\n",
    "\n",
    "http://api.worldbank.org/v2/countries/all/indicators/\n",
    "\n",
    "Here are some of the indicators/variables that can be extracted for World Bank's databases and appended to the above endpoint:\n",
    "\n",
    "'DC.DAC.TOTL.CD': 'net_aid',\n",
    "'EG.ELC.FOSL.ZS': 'percent_fossil_fuel',\n",
    "'EG.ELC.HYRO.ZS': 'percent_hydro',\n",
    "'EG.ELC.NUCL.ZS': 'percent_nuclear',\n",
    "'EG.ELC.RNWX.ZS': 'percent_renewable',\n",
    "'IT.NET.USER.P2': 'internet',\n",
    "'MS.MIL.XPND.ZS': 'mil_spend',\n",
    "'NY.GDP.PCAP.CD': 'GDP_per_capita',\n",
    "'SE.ADT.LITR.ZS': 'literacy',\n",
    "'SG.GEN.PARL.ZS': 'women_in_parliament',\n",
    "'SL.UEM.1524.ZS': 'youth_unemployment',\n",
    "'SM.POP.NETM'   : 'net_migration',\n",
    "'SP.DYN.IMRT.IN': 'infant_mortality',\n",
    "'SP.DYN.LE00.IN': 'life_expectancy',\n",
    "'SP.DYN.TFRT.IN': 'fertility',\n",
    "'SP.POP.GROW'   : 'population_growth',\n",
    "'VC.IHR.PSRC.P5': 'intentional_homicides'\n",
    "\n",
    "One of the key additional arguments that can be passed to the API query is the date range like:\n",
    "\n",
    "?date=2013:2015\n",
    "                     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# World Bank API - GDP example\n",
    "\n",
    "indicator =  'NY.GDP.PCAP.CD?date=2000:2018'\n",
    "url = \"http://api.worldbank.org/v2/countries/all/indicators/%s&format=json&per_page=5000\" % indicator\n",
    "print(url)\n",
    "response = requests.get(url)\n",
    "print(response)\n",
    "result = response.content\n",
    "result = json.loads(result)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The json object can then be converted into a dataframe :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "worldbank_df = pd.DataFrame.from_dict(result[1])\n",
    "worldbank_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to extract the country from the dict object in the country column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "worldbank_df['country'] = worldbank_df[['country']].applymap(lambda x : x['value'])\n",
    "worldbank_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "worldbank_df.country.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can next select and rename some of the columns into more meaningful names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "worldbank_df = worldbank_df[['country', 'countryiso3code', 'date', 'value']]\n",
    "worldbank_df.columns = ['country', 'countryiso3code', 'date', 'GDP_per_capita']\n",
    "worldbank_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Plot the GDP per capita for New Zealand from the above dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Perform the same steps for Singapore and plot the data against that of New Zealand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Generate a new API query that extract the net migration numbers for NZ, Singapore and Australia since the year 2000 then plot them together on the same graph.\n",
    "\n",
    "https://api.worldbank.org/v2/sources/40/indicators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  IMF API\n",
    "\n",
    "The INternational Monetary Fund provides an API for a comprehensive set of financial and economic indicators whose details can be explored here https://datahelpdesk.worldbank.org/knowledgebase/topics/125589-developer-information and http://datahelp.imf.org/knowledgebase/articles/838041-sdmx-2-0-restful-web-service\n",
    "\n",
    "\n",
    "There are some examples of how to extract data from this API in the following links:\n",
    "\n",
    "https://briandew.wordpress.com/2016/05/01/machine-reading-imf-data-data-retrieval-with-python/\n",
    "\n",
    "https://briandew.wordpress.com/2016/08/10/using-the-imf-data-api-data-retrieval-with-python/ \n",
    "\n",
    "https://www.bd-econ.com/imfapi1.html\n",
    "\n",
    "\n",
    "The endpoint for the IMF API service is http://dataservices.imf.org/REST/SDMX_JSON.svc/\n",
    "\n",
    "A number of different databases can then be appended to the endpoint; however, the documentation is somewhat obscure in terms of usage instructions. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is a request for quarterly (frequency: Q) import price index data (indicator: PMP_IX) for NZ. (reference area: NZ), \n",
    "# from the International Financial Statistics (IFS) series.\n",
    "\n",
    "url = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/' #endpoint\n",
    "database = 'CompactData/IFS/Q.NZ.PMP_IX' # database and indicator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(url + database)\n",
    "print(response)\n",
    "result = response.content\n",
    "result = json.loads(result)\n",
    "result\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['CompactData'].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['CompactData']['DataSet'].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['CompactData']['DataSet']['Series'].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['CompactData']['DataSet']['Series']['Obs']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.DataFrame.from_dict(result['CompactData']['DataSet']['Series']['Obs'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Clean up the dataframe and plot the annual percentage change in the NZ Export Price Index Inflation Rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Repeat the same analysis and plot for Australia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dedicated API Wrapper Modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well established companies will sometimes write and make available modules in various programming languages that form a wrapper around their REST APIs and an easier interface for communicating with their servers.\n",
    "\n",
    "Yahoo and Spotify are an example of such companies that provide a Python module. Some of custom-made APIs are free-access and some require an account to be created with them first. Premium content can only be pulled from their servers using a paid Premium account.\n",
    "\n",
    "Let's look at a custom wrapper for Craigslist:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install python-craigslist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from craigslist import CraigslistJobs, CraigslistForSale\n",
    "CraigslistJobs.show_filters()\n",
    "print(\"=========================\")\n",
    "CraigslistForSale.show_filters(category='cta')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find a software developer job in Auckland (codes have been worked out by studying the craigslist website URLs):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from craigslist import CraigslistJobs\n",
    "cl_j = CraigslistJobs(site='auckland', category='sof',\n",
    "                      filters={ 'employment_type': ['full-time', 'part-time']})\n",
    "\n",
    "for result in cl_j.get_results():\n",
    "    print(result)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Find a free food events in New York:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from craigslist import CraigslistEvents\n",
    "cl_e = CraigslistEvents(site='newyork', filters={'free': True, 'food': True})\n",
    "\n",
    "for result in cl_e.get_results(sort_by='newest', limit=5):\n",
    "    print(result)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API Repositories and Market Places\n",
    "\n",
    "A large number of other API repositories can be found under these links:\n",
    "\n",
    "https://any-api.com/\n",
    "\n",
    "https://market.mashape.com/\n",
    "\n",
    "A summary of some useful APIs can be found here http://www.computersciencezone.org/50-most-useful-apis-for-developers/ \n",
    "\n",
    "http://www.programmableweb.com/apis/directory\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
